Systems and methods for recommending physical activity associated with media content

ABSTRACT

Systems and methods are presented herein for providing a user with a recommendation for physical activity related to media content. The user may activate a recommendation application setting on a user equipment device. The user may select a media asset for presentation on a user equipment device. The system may analyze data associated with the media asset and perform deep video analysis of the frames of the media asset to identify events in the media asset. The system may identify the content type of each respective segment of the media asset and may identify respective activities performed in each segment by a primary character. The system may determine a respective activity to recommend to the user for reach respective segment of the media asset. The application may then determine a segment is being displayed for a user and generate an overlay representing a physical activity for the user to perform.

BACKGROUND

The present disclosure is directed to systems and methods for providinga recommendation for physical activity, and more particularly, tosystems and methods that generate a recommendation for physical activityfor the user based on events in media content.

SUMMARY

When a user is consuming media content provided by a content platform, auser may want to engage in a physical activity. In some instances, auser may have to find a separate application to enable alerts ornotifications to perform physical activities. This method is deficientin that content of these alerts or notifications will be limited to thecontent the user inputs into the alerts or reminders. Additionally, thealerts or notifications may not be synchronized with the media contentthe user is consuming, unless the user manually reviews the mediacontent and customizes alerts or reminders to reflect the subject matterof the media content. This method may also cause a user to lose interestin the media content in favor of pursuing a physical activity, as theuser may be distracted from the media content.

In another approach, a user may find an application that generatesexercise recommendations based on available exercises and workout plansloaded into the application's memory. This method is also deficient inthat the loaded content may not be associated with the subject matter ofthe media content and may cause a user to stop consuming the mediacontent in order to engage in the physical activity recommended by theapplication. Additionally, the user is limited to exercises preloaded inthe application and may not be able to customize recommendations toalign with media content the user has an interest in. The user may alsohave to stop consuming media content and search exercise recommendationsavailable through the application to create a customized set of exerciserecommendations.

These deficiencies are solved by the systems and methods describedherein, which in some embodiments provide a recommendation for physicalactivity for the user to perform based on a user's consumption of mediacontent, which may include a media content item. In some embodiments, asystem may have encoded on it a recommendation application. Therecommendation application may be capable of generating media contentfor presentation on a user equipment system. While the media content isgenerated for presentation, events may be determined from dataassociated with the media content that correspond to activitiesperformed by characters of interest in the media content. An activityfor the user to perform is determined based on the determined events ofthe media content and may be related to the events in the media content.A recommendation for the user to perform the determined activity may begenerated for presentation, proximate to the occurrence of the event inthe generated media content. The recommendation may consist of a graphicanimated overlay depicting the determined physical activity with awritten description of the activity.

These techniques solve the problems of the other approaches describedabove. In particular, the disclosed systems and methods do not require auser to stop consuming media content to pursue physical activity.Additionally, this approach does not require a user to manually programnotifications or alerts to start exercising, nor does this approachrequire a user to manually search for recommended exercises in adatabase to try to synchronize the recommended activities with eventsthat are occurring in the media content.

In some embodiments, the recommendation application may generate forpresentation the media content by generating the media content fordisplay on a system that the recommendation application is loaded onto.In some embodiments, the recommendation application may generate forpresentation the recommendation for physical activity by generating therecommendation for physical activity for display on a system that therecommendation application is loaded onto that is also displaying themedia content.

In some embodiments the recommendation application may identify an eventwithin media content by retrieving metadata associated with the mediacontent. In some embodiments, the recommendation application mayidentify an event within media content by analyzing video frame data inthe media content. In some embodiments, the recommendation applicationmay determine if other media content items available through a contentplatform have similar events as the first media content item selected.The recommendation application may utilize metadata and othercomputer-readable information about each respective media content itemavailable for consumption to determine primary characters and respectiveactivities based on event data retrieved or extracted from eachrespective media content item. The recommendation application may thengenerate for later presentation a series of recommended exercises andoverlays when the user selects each respective media content item forplayback.

In some embodiments, the recommendation application recognizes aplurality of events in media content as a plurality of respectivephysical activities performed by a character in the media content. Insome embodiments, the recommendation application may determine thatmultiple events that involve the same activity are in a same mediacontent item. In some embodiments, the recommendation application maydetermine there are a maximum number of times a same event will generatea recommendation in a particular media content item.

In some embodiments, the recommendation application may retrieve a userprofile associated with a request to generate media content items and arequest to generate recommendations for physical activity. Therecommendation application may review user preferences associated withthe user profile and may generate recommendations for physical activitybased on the user preferences associated with the user profile. In someembodiments, the recommendation application determines if the userprofile enabled the media recommendation setting within the user profilein response to a user input to enable the embodiments of therecommendation application described herein.

In some embodiments, the recommendation application generates overlaysthat may have an option for the user to close out the overlay before thesegment is over. Should a user close the overlay before the segment isover, the recommendation application may save that interaction as anindication that the recommended exercise does not suit the user'sinterest or capabilities and may remove that exercise from futurerecommendations.

In some embodiments, the recommendation application may identify otheruser interests and preferences based on the user's interaction withother content platforms and applications on the user equipment device.The recommendation application may utilize the collected preference datato generate new recommended activity overlays and proactively analyzeother media assets of related content types in anticipation that theuser may enable the application during the playback of other futuremedia assets.

In some embodiments, the recommendation application may review a firstevent and a second event in a media content item to determine a timedifference between the first event and the second event. Therecommendation application may compare the time difference to apredetermined threshold to determine whether to generate forpresentation a recommendation proximate to the occurrence of the secondevent.

In some embodiments, the recommendation may identify an event in a mediacontent item and may generate a prompt for the user to interact with todetermine if a subsequent recommendation for activity may be generatedduring the presentation of the identified event in the media contentitem.

BRIEF DESCRIPTIONS OF THE DRAWINGS

The above and other objects and advantages of the disclosure will beapparent upon consideration of the following detailed description, takenin conjunction with the accompanying drawings, in which:

FIG. 1 depicts an illustrative audio/video user entertainment system, inaccordance with some embodiments of the disclosure;

FIG. 2 depicts the recommendation application performing an illustrativeanalysis of frames of a media asset, in accordance with some embodimentsof the disclosure;

FIG. 3 is a block diagram representing devices, components of eachdevice, and the transfer of data therebetween for an audio/video userentertainment system incorporating physical activity recommendationfeatures, in accordance with some embodiments of the disclosure;

FIG. 4 is a flowchart of an illustrative process for generating arecommendation for physical activity that is related to media contentbeing consumed by a user, in accordance with disclosed methods andembodiments;

FIG. 5 illustrates an example of a generated overlay for a segment of amedia asset that provides a recommendation for physical activity, inaccordance with some embodiments of the disclosure;

FIG. 6 is a flowchart of an illustrative process for updating userpreferences based on a user's interaction with a generatedrecommendation for physical activity, in accordance with someembodiments of the disclosure;

FIG. 7 is a flowchart of an illustrative process for updating userpreferences based on a user's interaction with a generated promptrelated to a recommendation for physical activity, in accordance withsome embodiments of the disclosure;

FIG. 8 is a flowchart of an illustrative process for utilizing userpreferences to identify recommendations for physical activity, inaccordance with some embodiments of the disclosure.

DETAILED DESCRIPTION

Methods and systems are provided herein for providing a recommendationfor physical activity that is related to media content, which in someembodiments is a media content item, being consumed by a user.

The methods and/or any instructions for performing any of theembodiments discussed herein may be encoded on computer-readable media.Computer-readable media includes any media capable of storing data. Thecomputer-readable media may be transitory, including, but not limitedto, propagating electrical or electromagnetic signals, or may benon-transitory including, but not limited to, volatile and non-volatilecomputer memory or storage devices such as a hard disk, floppy disk, USBdrive, DVD, CD, media cards, register memory, processor caches, RandomAccess Memory (RAM), etc.

FIG. 1 depicts an illustrative audio/video user entertainment system100, in accordance with some embodiments of the disclosure. Theaudio/video user entertainment system 100 may include audio outputs,video display 101, interactive menus, a microphone for recording auser's commands for the system, and a device such as a remote control,which receives user input (e.g., a text input in a field in aninteractive interface to provide a command to the user entertainmentsystem). In some embodiments, a user may want to enable therecommendation application. A user may interact with video display 101,which in some embodiments generates a selectable settings menu 102 thata user can interact with to adjust settings associated with theaudio/video entertainment system 100. For example, selectable settingsmenu 102 may include activity recommendation setting 103 that includesan ON option 104 that a user can select. The application may recognizeON option 104 being selected by generating selected icon 105 that, whenenabled, provides an indication to the recommendation application toanalyze media assets to identify when to present activityrecommendations.

Video display 101 may generate a list of media assets available forviewing for a user to select. In some embodiments, one or more of themedia assets may be scheduled and available at a scheduled time. In someembodiments, one or more of the media assets may be available on aserver for a user to select to start viewing without the constraint of aschedule. Each media asset may have a selectable icon 106, which, whenselected, may inform the recommendation application that a particularmedia asset is selected for presentation. Each media asset may also haveassociated data 107, which in some embodiments enables therecommendation application to determine primary characters in the mediaasset whose activities will be tracked to generate physical activityrecommendations. For example, a user may select a movie such as“Mission: Impossible—Fallout,” which may include associated data thatindicates the actor Tom Cruise portrays a primary character in themovie. The recommendation application may then identify Tom Cruise assubject 111 who may be tracked throughout the frames of the media assetfor activities.

In some embodiments, the application analyzes the media asset by stepsthat may comprise identifying media asset segments (e.g., segment 108),analyzing the media asset frames per segment for elements, identifyingsubject 111 in each frame, and identifying the activity of subject 111.The application may then generate a recommended user activity bypresenting graphic icon 109 and descriptive text 110. In someembodiments, the media asset segments are identified by reviewing ofmetadata associated with the media content to determine if a series offrames comprises a scene within the media asset. In some embodiments,the media asset segments are identified by performing deep videoanalysis of the media asset frames to determine if a series of framescan be considered part of a scene (e.g., based on the activities ofprimary characters), which may then be identified as a segment. In someembodiments, the subject 111 may be identified by the applicationreviewing the metadata associated with the media content to determinewhich segments subject 111 is visible in. In some embodiments, subject111 may be identified by the application performing deep video analysisof each respective frame in a media asset to determine if a series offrames include subject 111. In some embodiments, the activity of subject111 may be identified by the application reviewing the metadataassociated with the media content to determine the activities subject111 is performing in each respective frame or segment. In someembodiments, the activity of subject 111 may be identified by theapplication utilizing deep video analysis to analyze each frame fordetectable activities being performed by subject 111.

In some embodiments, the recommended user activity may be determined bythe application comparing data extracted from each media asset segmentto a database of recommended physical activities available in storage(e.g., storage 316 of computing device 300 of FIG. 3) and thendetermining the recommendation based on a match between the dataassociated with a respective segment and a recommendation found in thedatabase. In some embodiments, the generation of a recommended useractivity comprises presenting graphic icon 109 and descriptive text 110may be performed by the application utilizing deep video analysis dataof the activity of subject 111 to generate graphic icon 109 in anoutline of subject 111 and animated to perform a character activityrelated to with descriptive text 110, which may be generated by theapplication based on metadata related to the activity of subject 111 inthe related frame or segment.

In some embodiments, a promotional overlay may also be generated whichis related to graphic icon 109 and descriptive text 110. In someembodiments, the promotional overlay may be related to browsing activityrelated to a user profile associated with the user who has selected themedia asset for presentation.

FIG. 2 depicts the recommendation application performing an illustrativeanalysis 200 of frames of a media asset. In some embodiments, analysis200 may start by identifying respective frames 201 a-d of a media assetselected for presentation, e.g., “Mission: Impossible—Fallout” featuringTom Cruise. In some embodiments, the application may be capable ofperforming deep video analysis that includes identifying respectivecharacters and character activities 202 a-d in each respective frame 201a-d. For example, the actor Tom Cruise may be seen fighting, jumping,running, and rock climbing in different scenes throughout the movie“Mission: Impossible—Fallout,” and the application may identify theseevents based on deep video analysis seeking figures that resemble TomCruise in each respective frame. In some embodiments, the applicationmay determine respective frame data 203 a-d (e.g., data that identifiesan Actor, an Environment, and an Event) for each respective frame 201a-d by utilizing metadata associated with the frames of the media asset.For example, metadata may identify actors, an environment, and anactivity occurring in each frame or series of frames.

In some embodiments, the application may utilize metadata associatedwith frames 201 a-d to identify respective segment element groups 204a-d, which may be used to define a segment within the media asset. Insome embodiments, the application may generate a different activityrecommendation (e.g., overlaid on the media content) for differentsegments of the movie. In some embodiments, the application may group aplurality of frames into a segment when the frames match each of theelements listed in one of respective segment element groups 204 a-d. Forexample, a respective segment element group may include an actor namethat may be Tom Cruise, an environment type that may be a rooftop, andan activity of the actor that may be running. In some embodiments, aframe may be analyzed by either a review of metadata or deep videoanalysis of the frame to determine the content of the frame. Forexample, a frame may have Tom Cruise running as identified in frame 201c and may have metadata represented by frame data 203 c, which includesidentifiers for the actor Tom Cruise in the environment of a rooftop andthe event where the actor is running. In this example, the depictedframe may match the elements of segment element group 204 c, andadditional frames with like data may also be grouped in the same segmentas frame 201 c.

In some embodiments, the application may use the elements of respectivesegment element groups 204 a-d to identify respective recommendedactivities 205 a-c. In some embodiments, the application may utilize theactivity identified in respective segment element groups 204 a-d, e.g.,running, as shown in segment element group 204 c. In some embodiments,the application may utilize the activity identified in respective framedata 203 a-d that is associated with the event, e.g., running, as shownin respective frame data 203 c. In some embodiments, the application mayutilize the results of deep video analysis of respective frames 202 a-d,e.g., Tom Cruise running in respective frame 202 c. The recommendedactivity generated in exemplary respective recommended activity 205 cmay be “Run in Place,” which may correlate to the respective data sourcethat was analyzed for generating the recommendation. The application mayutilize any of the described analysis options alone or in combination,depending on the processing capabilities enabled on the system on whichthe recommendation application is implemented.

In some embodiments, the application is capable of accessing contentserver 302, e.g., as shown in FIG. 3. The application is capable ofutilizing the processing of multiple modules connected by communicationnetwork 306 to find and review the information available on contentserver 302.

FIG. 3 is an illustrative block diagram showing an interactiveaudio/video system for providing a recommendation for physical activityfor the user based on events in media content, in accordance with someembodiments of the disclosure. In some embodiments, one or more parts ofor the entirety of the interactive system may be configured as a systemimplementing various features, processes, and components of FIGS. 1, 2,and 4-8. Although FIG. 3 shows a certain number of components, invarious examples, the interactive system may include fewer than theillustrated number of components and/or multiples of one or more of theillustrated number of components.

The interactive system is shown to include computing device 300, contentserver 302, and a communication network 306. It is understood that whilea single instance of a component may be shown and described relative toFIG. 3, additional instances of the component may be employed. Forexample, content server 302 may include, or may be incorporated in, morethan one server. Similarly, communication network 306 may include, ormay be incorporated in, more than one communication network. Contentserver 302 is shown communicatively coupled to computing device 300through communication network 306. While not shown in FIG. 3, contentserver 302 may be directly communicatively coupled to computing device300, for example, in a system absent or bypassing communication network306.

Communication network 306 may comprise one or more network systems, suchas, without limitation, Internet, LAN, WiFi or other network systemssuitable for audio processing applications. In some embodiments, thesystem of FIG. 3 excludes content server 302, and functionality thatwould otherwise be implemented by content server 302 is insteadimplemented by other components of the system depicted by FIG. 3, suchas one or more components of communication network 306. In still otherembodiments, content server 302 may work in conjunction with one or morecomponents of communication network 306 to implement certainfunctionality described herein in a distributed or cooperative manner.Similarly, in some embodiments, the system depicted by FIG. 3 excludescomputing device 300, and functionality that would otherwise beimplemented by computing device 300 is instead implemented by othercomponents of the system depicted by FIG. 3, such as one or morecomponents of communication network 306 or content server 302 or acombination of the same. In other embodiments, computing device 300works in conjunction with one or more components of communicationnetwork 306 or content server 302 to implement certain functionalitydescribed herein in a distributed or cooperative manner.

Computing device 300 includes control circuitry 308, display circuitry310 and input/output circuitry 312. Control circuitry 308 may be basedon any suitable processing circuitry and comprises control circuits andmemory circuits, which may be disposed on a single integrated circuit ormay be discrete components. As referred to herein, processing circuitryshould be understood to mean circuitry based on one or moremicroprocessors, microcontrollers, digital signal processors,programmable logic devices, field-programmable gate arrays (FPGAs),application-specific integrated circuits (ASICs), etc., and may includea multi-core processor (e.g., dual-core, quad-core, hexa-core, or anysuitable number of cores). In some embodiments, processing circuitry maybe distributed across multiple separate processors or processing units,for example, multiple of the same type of processing units (e.g., twoIntel Core i7 processors) or multiple different processors (e.g., anIntel Core i5 processor and an Intel Core i7 processor). Some controlcircuits may be implemented in hardware, firmware, or software. Controlcircuitry 308 in turn includes transceiver circuitry 314, storage 316and processing circuitry 318. In some embodiments, computing device 300or control circuitry 308 may be configured as varying embodiments ofaudio/video user entertainment system 100 of FIG. 1.

In addition to control circuitry 308 and 320, computing device 300,content server 302, may each include storage (storage 316 and storage322, respectively). Each of storages 316 and 322 may be an electronicstorage device. As referred to herein, the phrase “electronic storagedevice” or “storage device” should be understood to mean any device forstoring electronic data, computer software, or firmware, such asrandom-access memory, read-only memory, hard drives, optical drives,digital video disc (DVD) recorders, compact disc (CD) recorders, BLU-RAYdisc (BD) recorders, BLU-RAY 3D disc recorders, digital video recorders(DVRs, sometimes called personal video recorders, or PVRs), solid statedevices, quantum storage devices, gaming consoles, gaming media, or anyother suitable fixed or removable storage devices, and/or anycombination of the same. Each storage 316 and 322 may be used to storevarious types of content, metadata, and/or other types of data (e.g.,they can be used to record audio questions asked by one or moreparticipants connected to a conference). Non-volatile memory may also beused (e.g., to launch a boot-up routine and other instructions).Cloud-based storage may be used to supplement storages 316 and 322 orinstead of storages 316 and 322. In some embodiments, the user profileactivity history, user profile preferences, and accessible media contentmay be stored in one or more of storages 316 and 322.

In some embodiments, control circuitry 320 and/or 308 executesinstructions for an application stored in memory (e.g., storage 322and/or storage 316). Specifically, control circuitry 320 and/or 308 maybe instructed by the application to perform the functions discussedherein. In some implementations, any action performed by controlcircuitry 320 and/or 308 may be based on instructions received from theapplication. For example, the application may be implemented as softwareor a set of executable instructions that may be stored in storage 322and/or 316 and executed by control circuitry 320 and/or 308. In someembodiments, the application may be a client/server application whereonly a client application resides on computing device 300, and a serverapplication resides on content server 302.

The application may be implemented using any suitable architecture. Forexample, it may be a stand-alone application wholly implemented oncomputing device 300. In such an approach, instructions for theapplication are stored locally (e.g., in storage 316), and data for useby the application is downloaded on a periodic basis (e.g., from anout-of-band feed, from an Internet resource, or using another suitableapproach). Control circuitry 308 may retrieve instructions for theapplication from storage 316 and process the instructions to perform thefunctionality described herein. Based on the processed instructions,control circuitry 308 may determine to execute elements of theembodiments of this disclosure in response to input received frominput/output circuitry 312 or from communication network 306. Forexample, in response to a user providing inputs to activateentertainment system 100, control circuitry 308 may perform the steps ofany of the processes depicted in FIGS. 1, 2, and 4-8, or processesrelative to various embodiments.

In client/server-based embodiments, control circuitry 308 may includecommunication circuitry suitable for communicating with an applicationserver (e.g., content server 302) or other networks or servers. Theinstructions for carrying out the functionality described herein may bestored on the application server. Communication circuitry may include acable modem, an Ethernet card, or a wireless modem for communicationwith other equipment, or any other suitable communication circuitry.Such communication may involve the Internet or any other suitablecommunication networks or paths (e.g., communication network 306). Inanother example of a client/server-based application, control circuitry308 runs a web browser that interprets web pages provided by a remoteserver (e.g., content server 302). For example, the remote server maystore the instructions for the application in a storage device. Theremote server may process the stored instructions using circuitry (e.g.,control circuitry 308) and/or generate displays. Computing device 300may receive the displays generated by the remote server and may displaythe content of the displays locally via display circuitry 310. This way,the processing of the instructions is performed remotely (e.g., bycontent server 302) while the resulting displays, such as the displaywindows described elsewhere herein, are provided locally on computingdevice 300. Computing device 300 may receive inputs from the user viainput/output circuitry 312 and transmit those inputs to the remoteserver for processing and generating the corresponding displays.Alternatively, computing device 300 may receive inputs from the user viainput/output circuitry 312 and process and display the received inputslocally, by control circuitry 308 and display circuitry 310,respectively.

Content server 302 and computing device 300 may transmit and receivecontent and data such as media content via communication network 306.For example, content server 302 may be a media content provider, andcomputing device 300 may be a smart television configured to download orstream media content, such as a live news broadcast, from content server302. Control circuitry 320, 308 may send and receive commands, requests,and other suitable data through communication network 306 usingtransceiver circuitry 342, 314, respectively. Control circuitry 320, 308may communicate directly with each other using transceiver circuitry342, 314, respectively, avoiding communication network 306.

It is understood that computing device 300 is not limited to theembodiments and methods shown and described herein. In nonlimitingexamples, computing device 300 may be a television, a Smart TV, aset-top box, an integrated receiver decoder (IRD) for handling satellitetelevision, a digital storage device, a digital media receiver (DMR), adigital media adapter (DMA), a streaming media device, a DVD player, aDVD recorder, a connected DVD, a local media server, a BLU-RAY player, aBLU-RAY recorder, a personal computer (PC), a laptop computer, a tabletcomputer, a WebTV box, a personal computer television (PC/TV), a PCmedia server, a PC media center, a handheld computer, a stationarytelephone, a personal digital assistant (PDA), a mobile telephone, aportable video player, a portable music player, a portable gamingmachine, a smartphone, or any other device, computing equipment, orwireless device, and/or combination of the same capable of suitablydisplaying and manipulating media content.

Control circuitry 320 and/or 308 may be based on any suitable processingcircuitry such as processing circuitry 328 and/or 318, respectively. Asreferred to herein, processing circuitry should be understood to meancircuitry based on one or more microprocessors, microcontrollers,digital signal processors, programmable logic devices,field-programmable gate arrays (FPGAs), application-specific integratedcircuits (ASICs), etc., and may include a multi-core processor (e.g.,dual-core, quad-core, hexa-core, or any suitable number of cores). Insome embodiments, processing circuitry may be distributed acrossmultiple separate processors, for example, multiple of the same type ofprocessors (e.g., two Intel Core i9 processors) or multiple differentprocessors (e.g., an Intel Core i7 processor and an Intel Core i9processor). In some embodiments, control circuitry 320 and/or controlcircuitry 308 are configured to implement a media content operationsystem, such as systems, or parts thereof, that perform variousprocesses described and shown in connection with FIGS. 1, 2, and 4-8,and/or systems carrying out the features described and shown relative toFIGS. 1, 2, and 4-8.

Computing device 300 receives user input 332 at input/output circuitry312. For example, computing device 300 may receive a user input such asa user swipe or user touch, as previously discussed. In someembodiments, computing device 300 is a media device (or player)configured as entertainment system 100, with the capability to accessmedia content. It is understood that computing device 300 is not limitedto the embodiments and methods shown and described herein. Innonlimiting examples, computing device 300 may be a television, a SmartTV, a set-top box, an integrated receiver decoder (IRD) for handlingsatellite television, a digital storage device, a digital media receiver(DMR), a digital media adapter (DMA), a streaming media device, a DVDplayer, a DVD recorder, a connected DVD, a local media server, a BLU-RAYplayer, a BLU-RAY recorder, a personal computer (PC), a laptop computer,a tablet computer, a WebTV box, a personal computer television (PC/TV),a PC media server, a PC media center, a handheld computer, a stationarytelephone, a personal digital assistant (PDA), a mobile telephone, aportable video player, a portable music player, a portable gamingmachine, a smartphone, or any other television equipment, computingequipment, or wireless device, and/or combination of the same.

User input 332 may be received from a user selection-capturing interfacethat is separate from device 300, such as a remote-control device,trackpad or any other suitable user movement sensitive or capturedevices, or as part of device 302, such as a touchscreen of displaycircuitry 310. Transmission of user input 332 to computing device 300may be accomplished using a wired connection, such as an audio cable,USB cable, ethernet cable or the like attached to a corresponding inputport at a local device, or may be accomplished using a wirelessconnection, such as Bluetooth, WiFi, WiMAX, GSM, UTMS, CDMA, TDMA, 3G,4G, 4G LTE, 5G, or any other suitable wireless transmission protocol.Input/output circuitry 312 may comprise a physical input port such as a3.5 mm audio jack, RCA audio jack, USB port, ethernet port, or any othersuitable connection for receiving audio over a wired connection, or maycomprise a wireless receiver configured to receive data via Bluetooth,WiFi, WiMAX, GSM, UTMS, CDMA, TDMA, 3G, 4G, 4G LTE, 5G, or otherwireless transmission protocols.

Processing circuitry 318 may receive user input 332 from input/outputcircuitry 312 using communication path 334. Processing circuitry 318 mayconvert or translate the received user input 332 which may be in theform of gestures or movement, to digital signals. In some embodiments,input/output circuitry 312 performs the translation to digital signals.In some embodiments, processing circuitry 318 (or processing circuitry328, as the case may be) carries out disclosed processes and methods.For example, processing circuitry 318 or processing circuitry 328 mayperform processes of FIGS. 1, 2, and 4-8, respectively.

Processing circuitry 318 may provide requests to storage 316 bycommunication path 336. Storage 316 may provide requested information toprocessing circuitry 318 by communication path 338. Storage 316 maytransfer, by communication path 338, a request for information totransceiver circuitry 314, which may translate or encode the request forinformation to a format receivable by communication network 306 beforetransferring the request for information by communication path 340.Communication network 306 may forward the translated or encoded requestfor information to transceiver circuitry 342 by communication path 346.

At transceiver circuitry 342, the translated or encoded request forinformation, received through communication path 346, is translated ordecoded for processing circuitry 328, which will provide a response tothe request for information (e.g., additional activities associated withan event) based on information available through control circuitry 320or storage 322, or a combination thereof. The response to the requestfor information is then provided back to communication network 306 bycommunication path 350 in an encoded or translated format such thatcommunication network 306, which can forward the encoded or translatedresponse back to transceiver circuitry 314 by communication path 352.

At transceiver circuitry 314, the encoded or translated response to therequest for information may be provided directly back to processingcircuitry 318 by communication path 356, or may be provided to storage316, through communication path 358, which then provides the informationto processing circuitry 318 by communication path 360. Processingcircuitry 318 may also provide a request for information directly totransceiver circuitry 314 though communication path 362, enablingstorage 316 to respond to an information request, provided throughcommunication 336, by communication path 360 that storage 316 does notcontain information pertaining to the request from processing circuitry318.

Processing circuitry 318 may process the response to the requestreceived through communication path 356 or 360 and may provideinstructions to display circuitry 310 for a notification to be providedto the users through communication path 364. Display circuitry 310 mayincorporate a timer for providing the notification or may rely on inputsthrough input/output circuitry 312 from the user, which are forwardedthrough processing circuitry 318 through communication path 364, todetermine how long or in what format to provide the notification. Whendisplay circuitry 310 determines the display has been completed (e.g.,media content has completed a playback time or a user has exited out ofa recommendation), a notification may be provided to processingcircuitry 310 through communication path 366.

The communication paths provided in FIG. 3 between computing device 300,content server 302, communication network 306, and all subcomponentsdepicted are exemplary and may be modified to reduce processing time orenhance processing capabilities for each step in the processes disclosedherein by one skilled in the art.

FIG. 4 is a flowchart of illustrative process 400 for generating arecommendation for physical activity that is related to media contentbeing consumed by a user, in accordance with some embodiments of thedisclosure. Process 400 or any step thereof could be performed on, orprovided by, any device shown in FIG. 3 and can incorporate various userinterfaces (e.g., display 310 of FIG. 3). For example, process 400 maybe executed by control circuitry 308 (FIG. 3) of computing device 300.In addition, one or more steps of process 400 may be incorporated intoor combined with one or more steps of any other process or embodiment(e.g., process block 601 of process 600 depicted in FIG. 6, processblock 701 of process 700 depicted in FIG. 7, or process block 801 ofprocess 800 depicted in FIG. 8). FIGS. 1, 2, and 5 provide depictions ofthe execution of process 400.

At 401, the application may receive a user request to provide arecommendation for a user activity. In some embodiments, a user mayinteract with an interactive menu and select an icon to enable theapplication. In some embodiments, the user may speak a command to a userequipment device capable of receiving and processing audio user commandsto enable the applications. In some embodiments, 401 is optional and theapplication may default to providing recommendations for user activity.

At 402, the application may generate media content for presentation tothe user (e.g., using display 310 of FIG. 3). In some embodiments, themedia content is generated for display to the user. In some embodiments,the media content is audibly generated for presentation to the user. Insome embodiments, the media content is generated for presentation inresponse to a user selection of the media asset.

At 403, the application may identify an event in the media content. Theevent may, for example, be associated with a physical activity beingpresented in the media content as being enacted by a character in themedia content. In some embodiments, the application identifies a primarycharacter based on metadata associated with the media asset and trackseach frame for that primary character. In some embodiments, theapplication associates a primary character with a figure that isdetected in a majority of the frames of the media asset through deepvideo analysis. The application may associate each respective segment toa content type, based on the primary activity detected in each segmentof the media asset. In some embodiments, the media content includespredetermined metadata associated with particular events that, whendetected, may notify the application of a recommended physical activity.

In some embodiments, there may be metadata associated with particularframes and corresponding segments that may indicate that the event doesnot last long enough during a particular segment of media content togenerate a recommendation to a user. In some embodiments, the metadataassociated with a particular event may not identify an activity that isassociated with user preferences in a user profile and, as a result, thesystem may not flag that event for generating a recommendation. In someembodiments, a plurality of instances of the same activity may appear inthe metadata of the media content for a plurality of segments. In someembodiments, the application may incorporate a threshold number ofmaximum instances of recommending a particular activity within mediacontent and may limit the number of recommendations generated for aparticular event type within a particular media content item.

At 404, the application determines an activity to recommend to the userbased on the identified event. In some embodiments, determining arecommended activity may be accomplished by the application comparingdata extracted from media asset segments to a preloaded database ofrecommended activities available in, for example, storage 316 ofcomputing device 300 of FIG. 3. The recommendation activity may then bedetermined based on a match between the data associated with arespective segment and a recommendation found in the preloaded database.In some embodiments, determining a recommended activity may include theapplication utilizing deep video analysis data of the activity of thesubject, e.g. subject 111 of FIG. 1 to generate an overlay icon that isan outline of the subject animated to perform the recommended activity.In some embodiments, determining an activity includes determiningrelated promotional content that is related to the recommended activityand past user activity monitored on a user equipment device.

At 405, the application generates for presentation the recommendationfor performing the activity associated with the segment by the analysisperformed at 403 and at 404. In some embodiments, the recommendationincludes an overlay that is comprised of a written description, agraphic display of an icon demonstrating the physical activity, and anarea that incorporates promotional content. In some embodiments, therecommendation is a prompt that is generated, as a first overlay, forpresentation prior to the event occurring in the media content, and theprompt may request a user input to determine if a second overlay with ananimation incorporating the recommendation may be displayed for the userto interact with.

FIG. 5 illustrates an example of overlaid media content 500 for asegment of a media asset that provides a recommendation for physicalactivity, in accordance with some embodiments of the disclosure. Thegenerated overlay may be provided by any device shown in FIG. 3 and canincorporate various user interfaces (e.g., display 310 of FIG. 3). Forexample, process 400 may be executed by control circuitry 308 (FIG. 3)of computing device 300 and may generate overlaid media content 500. Inaddition, one or more steps of process 400 may be incorporated into orcombined with one or more steps of any other process or embodiment(e.g., process block 601 of process 600 depicted in FIG. 6) and maygenerate overlaid media content 500. FIGS. 1 and 2 provide depictions ofthe execution of process 400 with an example of overlay, in accordancewith some embodiments of the disclosure.

Overlaid media content 500 may be generated by the application after theapplication performs a review of frame 501 of a media content item todetermine the placement of sub-content area 503 within frame 501. Insome embodiments, the application may utilize frame activity area 502 todetermine where the viewer may have interest in frame 501 when a mediaasset is selected for presentation. In some embodiments, the applicationdetermines frame activity area 502 by deep video analysis, which mayinclude identifying a character of interest, in accordance with someembodiments of the disclosure. In some embodiments, the application mayutilize metadata, which may provide descriptions of what is displayed ineach frame, to identify frame activity area 502. The application maydefine criteria for the amount of a frame that activity area 502 maycover, e.g., a threshold number of pixels, and may incorporate anexception that sub-content area 503 may overlap onto frame activity area502 when frame activity area 502 encompasses a percentage frame areathat is greater than a threshold.

Sub-content area 503 may be populated by animated recommendation 504,recommendation description 505, and promotional content 506. In someembodiments, sub-content area 503 may incorporate exit indicator 507which, when selected, may close sub-content area 503 prior to thecompletion of the segment of the media asset sub-content area 503 wasgenerated for. Animated recommendation 504 may be generated byretrieving information associated with analysis 200 in FIG. 2 which maybe found in computing device 300 or content server 302 throughcommunication network 306. In some embodiments, animated recommendation504 may include a graphic animation representing the activity the usermay perform for the duration of the related segment of the media asset.In some embodiments, animated recommendation 504 may be an icon thatdepicts the activity the user may perform for the duration of therelated segment of the media asset. Recommendation description 505 maybe generated by retrieving information associated with analysis 200 inFIG. 2, which may be found in computing device 300 or content server 302through communication network 306. In some embodiments, recommendationdescription 505 may be a command associated with the activitydemonstrated by animated recommendation 504. Promotional content 506 maybe generated by retrieving information associated with analysis 200 inFIG. 2 that may be found in computing device 300 or content server 302through communication network 306. In some embodiments, promotionalcontent 506 may be associated with any or all of a user profile viewinghistory, user profile activity retrievable on computing device 300 orcontent server 302 through communication network 306, or any of theelements extracted during analysis 200 of FIG. 2.

FIG. 6 is a flowchart of illustrative process 600 for updating userpreferences based on a user's interaction with a generatedrecommendation for physical activity, in accordance with someembodiments of the disclosure. Process 600 or any step thereof could beperformed on, or provided by, any device shown in FIG. 3 and canincorporate various user interfaces (e.g., display 310 of FIG. 3). Forexample, process 600 may be executed by control circuitry 308 (FIG. 3)of computing device 300. In addition, one or more steps of process 600may be incorporated into or combined with one or more steps of any otherprocess or embodiment (e.g., process block 405 of process 400 depictedin FIG. 4, process block 701 of process 700 depicted in FIG. 7, orprocess block 801 of process 800 depicted in FIG. 8). FIGS. 1 and 2provide depictions of the execution of process 600.

At 601, the application generates a recommendation of a respectiveactivity associated with a segment of a media asset being presented tothe user. In some embodiments, the overlay is generated using analysis200 in FIG. 2. In some embodiments, the recommendation includes overlaidmedia content 500 shown in FIG. 5. For example, overlaid media content500 incorporates multiple interactive elements such as promotionalcontent 506 or exit indicator 507 or a combination of both. In someembodiments, the application may utilize motion sensors, cameras, orother forms of detection of user activity to determine if the user isengaging in the recommended activity. For example, input/outputcircuitry 312 of FIG. 3 may incorporate a plurality of user detectionequipment to provide data to computing device 300 indicating that a useris engaged with the media content and the application. If theapplication determines that the user engaged in the recommended activitybefore the segment playback was completed (YES at 602), the applicationmay proceed to record in the user profile history that the user acceptedand enacted the recommended activity at 603. If the applicationdetermines that the user did not enact the activity in therecommendation (NO at 602), the application may update user profilepreferences to exclude the recommended activity at 604.

At 605, the application may search a server for a new recommendedactivity associated with identified media asset data. In someembodiments, the application may determine the device used to play backthe media asset is connected to content server 302 of FIG. 3 which maybe connected to the user device by communication network 306, inaccordance with some embodiments in the disclosure. At 606, theapplication may associate a new recommended activity to make part ofoverlaid media content 500, based on the results of the search at 605.In some embodiments, the application may update user profile preferencesassociated with the identified media asset data to include the newrecommended activity associated at 606.

FIG. 7 is a flowchart of an illustrative process for updating userpreferences based on a user's interaction with a generated promptrelated to a recommendation for physical activity, in accordance withsome embodiments of the disclosure. Process 700 or any step thereofcould be performed on, or provided by, any device shown in FIG. 3 andcan incorporate various user interfaces (e.g., display 310 of FIG. 3).For example, process 700 may be executed by control circuitry 308 (FIG.3) of computing device 300. In addition, one or more steps of process700 may be incorporated into or combined with one or more steps of anyother process or embodiment (e.g., process block 405 of process 400depicted in FIG. 4, process block 601 of process 600 depicted in FIG. 6,or process block 801 of process 800 depicted in FIG. 8). FIGS. 1 and 2provide depictions of the execution of process 700.

At 701, the application may generate a prompt for the user to engage arecommended activity associated with an upcoming segment of mediacontent that may be presented to the user. In some embodiments, theapplication may utilize a predetermined period of time to determine whento generate the prompt prior to the event being generated for display tothe user. In some embodiments, the application may determine that anupcoming segment incorporates an event that is associated with arecommended activity previously queued for the user. The applicationmay, in response to determining a user recently engaged in the relatedactivity, determine the upcoming segment will not be queued forgenerating a recommendation. In some embodiments, the application maygenerate the prompt utilizing the analysis to generate the overlaydepicted in FIG. 5 to ensure the prompt does not obscure the mediacontent that is generated for presentation to the user.

If the application determines that the user interacted with the promptby indicating the user wants to engage the recommendation (“YES” at702), the application may record in the activity history of the userprofile that the user chose to engage the recommendation by viewing theoverlay at 703. At 704, the application may generate the recommendedoverlay when the segment containing metadata associated with the eventthat occurs within that segment of the media content is generated forpresentation for the user. In some embodiments, the application may relyon the analysis depicted in FIGS. 1, 2, and 4 to determine the contentof the overlay to display. In some embodiments, the applicationgenerates the recommendation utilizing the overlay depicted in FIG. 5.

If the application determines that the user did not decide to interactwith the prompt by indicating the user wants to engage therecommendation (“NO” at 702), the application may check at 705 if theuser selected a “NO” indication in the prompt. If the applicationdetermines the user decided to not interact with the prompt byindicating the does not want to engage the recommendation (“YES” at705), the application may remove the prompt from the generated displayof the media content at 706. At 707, the application may update the userprofile preferences associated with media asset data to exclude therecommended activity that was rejected by the user. If the applicationdetermines the user decided to not interact with the prompt byindicating the does not want to engage the recommendation (“NO” at 705),the application may check if the prompt countdown timer expired at 708.In some embodiments, the application has a predetermined amount of timethat a prompt may be generated for display. The application may generatethe prompt for display a predetermined amount of time before an eventand the prompt may remain for display until a user interacts with theprompt or the timer expires. If the application determines the promptcountdown timer expired (“YES” at 708), the application may remove theprompt from the display at 706. If the application determines the promptcountdown timer did not expire (“NO” at 708), the application maycontinue generating the prompt until the application determines the userinteracted with the prompt or the application determines the promptcountdown timer expired.

FIG. 8 is a flowchart of an illustrative process for utilizing userpreferences to identify recommendations for physical activity, inaccordance with some embodiments of the disclosure. Process 800 or anystep thereof could be performed on, or provided by, any device shown inFIG. 3 and can incorporate various user interfaces (e.g., display 310 ofFIG. 3). For example, process 800 may be executed by control circuitry308 (FIG. 3) of computing device 300. In addition, one or more steps ofprocess 800 may be incorporated into or combined with one or more stepsof any other process or embodiment (e.g., process block 405 of process400 depicted in FIG. 4, process block 601 of process 600 depicted inFIG. 6, or process block 701 of process 700 depicted in FIG. 7).

At 801, the application may determine an event is in a segment of mediacontent. The application may utilize analysis 200 depicted in FIG. 2 orany other analysis one familiar with the art of the present disclosuremay implement. At 802, the application may identify elements representedby data extracted utilizing a form of data analysis (e.g., analysis 200)that are associated with the event (e.g., actors in a frame, environmentof the scene of the frame, event involving actors). At 803, theapplication may identify a user profile associated with a user requestto provide a recommendation for a user activity and associated withmedia content to be generated for presentation. At 804, the applicationmay identify user preferences related to user activities associated withthe identified user profile. At 805, the application may compare theuser preferences related to activities to the identified element data ofan event identified in selected media content. If the applicationdetermines the element data matches at least one user preference (“YES”at 806), the application may generate for presentation a recommendationfor performing the activity proximate to the occurrence of the event inthe media content. For example, the recommendation can be presented apredetermined amount of time before the occurrence of the event and/orsimultaneously with the occurrence of the event. If the applicationdetermines the element data does not match at least one user preference(“NO” at 806), the application may update the user preferences to notgenerate a recommendation for this media content and may update the userpreferences to avoid generating a recommendation with future relatedevents detected by the application in future media content selected forgeneration by the user profile.

The systems and processes discussed above are intended to beillustrative and not limiting. One skilled in the art would appreciatethat the actions of the processes discussed herein may be omitted,modified, combined, and/or rearranged, and any additional actions may beperformed without departing from the scope of the invention. Moregenerally, the above disclosure is meant to be exemplary and notlimiting. Only the claims that follow are meant to set bounds as to whatthe present disclosure includes. Furthermore, it should be noted thatthe features and limitations described in any one embodiment may beapplied to any other embodiment herein, and flowcharts or examplesrelating to one embodiment may be combined with any other embodiment ina suitable manner, done in different orders, or done in parallel. Inaddition, the systems and methods described herein may be performed inreal time. It should also be noted that the systems and/or methodsdescribed above may be applied to, or used in accordance with, othersystems and/or methods.

While some portions of this disclosure may refer to “convention” orexamples, any such reference is merely to provide context to the instantdisclosure and does not form any admission as to what constitutes thestate of the art.

1. A method for providing activity recommendations, comprising:generating, via a content server, for presentation a media content item;identifying, via the content server, an event in the media content item;determining, via the content server based on the event, an activity tobe performed; and generating, via the content server, for presentationat a client device a recommendation for performing the activityproximate to the occurrence of the event in the generated media contentitem.
 2. The method of claim 1, wherein the generating for presentationthe media content item comprises generating the media content item fordisplay on a display.
 3. The method of claim 2, wherein the generatingfor presentation the recommendation comprises generating therecommendation for display on the display.
 4. The method of claim 1,wherein the identifying the event comprises retrieving metadataassociated with the media content item.
 5. The method of claim 1,wherein the identifying the event comprises analyzing video frame datain the media content item.
 6. The method of claim 1, wherein the eventcomprises a physical activity performed by a character in the mediacontent item.
 7. The method of claim 1, further comprising: retrieving auser profile; determining to provide media recommendations based on theuser profile; and in response to determining to provide mediarecommendations based on the user profile, generating for presentationthe recommendation for performing the activity.
 8. The method of claim7, wherein the user profile comprises a media recommendation setting andwherein the media recommendation setting is set in response to receivinga user input.
 9. The method of claim 1, wherein the event comprises afirst event, the method further comprising: identifying a secondsubsequent event in the media content item; determining a timedifference between the first event and the second subsequent event;comparing the time difference to a predetermined threshold; and inresponse to comparing the time difference to the predeterminedthreshold, determining whether to generate for presentation arecommendation for performing an activity proximate to the occurrence ofthe second subsequent event.
 10. The method of claim 1, wherein: theevent in the media content item comprises an event type; determining theactivity to be performed comprises: comparing the event type to eventtypes of a plurality of activities to be performed; and identifying amatch based on the comparing.
 11. A system comprising: input/outputcircuitry configured to: generate for presentation a media content item;control circuitry configured to: identify an event in the media contentitem; determine, based on the event, an activity to be performed; andgenerate for presentation a recommendation for performing the activityproximate to the occurrence of the event in the generated media contentitem.
 12. The system of claim 11, wherein the input/output circuitry isconfigured to generate for presentation the media content item bygenerating the media content item for display on a display.
 13. Thesystem of claim 12, wherein the input/output circuitry is configured togenerate for presentation the recommendation by generating therecommendation for display on the display.
 14. The system of claim 11,wherein the control circuitry is configured to identify the event byretrieving metadata associated with the media content item.
 15. Thesystem of claim 11, wherein the control circuitry is configured toidentify the event by analyzing video frame data in the media contentitem.
 16. The system of claim 11, wherein the event comprises a physicalactivity performed by a character in the media content item.
 17. Thesystem of claim 11, wherein the control circuitry is further configuredto: retrieve a user profile; determine to provide media recommendationsbased on the user profile; and in response to determining to providemedia recommendations based on the user profile, generate forpresentation the recommendation for performing the activity.
 18. Thesystem of claim 17, wherein the user profile comprises a mediarecommendation setting and wherein the media recommendation setting isset in response to receiving a user input.
 19. The system of claim 11,wherein the event comprises a first event, and wherein the controlcircuitry is further configured to: identify a second subsequent eventin the media content item; determine a time difference between the firstevent and the second subsequent event; compare the time difference to apredetermined threshold; and in response to comparing the timedifference to the predetermined threshold, determine whether to generatefor presentation a recommendation for performing an activity proximateto the occurrence of the second subsequent event.
 20. The system ofclaim 11, wherein: the event in the media content item comprises anevent type; and the control circuitry is configured to determine theactivity to be performed by: comparing the event type to event types ofa plurality of activities to be performed; and identifying a match basedon the comparing. 21.-30. (canceled)